Add ComposedNamedPhysicalExtensionCodec by gabotechs · Pull Request #24826 · apache/datafusion

gabotechs · 2026-08-31T17:03:17Z

Which issue does this PR close?

Closes #.

Rationale for this change

The ComposedPhysicalExtensionCodec allowed to register multiple codecs, which are tried one by one until one works. When one works, the index of the list that occupies the codec that works is stored in the encoded payload for later decoding:

## Encoding

[
  MyFirstCodec,  <- tried, but did not work
  MySecondCodec, <- tried, but did not work
  MyThirdCodec, <- tried, and it work, store index 2 in payload
  MyFourthCodec
]

payload { index: 2, data: ... }

## Decoding

payload { index: 2, data: ... }

[
  MyFirstCodec,
  MySecondCodec,
  MyThirdCodec, <- index is 2, just this codec is attempted
  MyFourthCodec
]

The issue with this setup is that codecs can only be added to the end of the list, otherwise, while decoding, the index will be messed up and decoding will fail. In a setup with progressive rollouts this is very likely to happen.

While working in a small team, it's reasonable to expect developers to behave and just add new codecs at the end, however, when there are many teams contributing to the same list, with a non-centralized piece of code that has mutable access to it, it's easy to be in the situation where new codecs are not added at the end.

What changes are included in this PR?

Introduces a new ComposedNamedPhysicalExtensionCodec that identifies encoded payloads by codec name instead of by index in a list. This way, registration order does not matter, and this problem becomes irrepresentable:

## Encoding

{
  "MyFirstCodec": MyFirstCodec,  <- tried, but did not work
  "MySecondCodec": MySecondCodec, <- tried, but did not work
  "MyThirdCodec": MyThirdCodec, <- it worked!
  "MyFourthCodec": MyFourthCodec
}

payload { name: "MyThirdCodec", data: ... }

## Decoding

payload { index: "MyThirdCodec", data: ... }

{
  "MyFourthCodec": MyFourthCodec,
  "MyThirdCodec": MyThirdCodec, <- use this one, no matter the order
  "MyFirstCodec": MyFirstCodec,
  "MySecondCodec": MySecondCodec
}

What is the testing strategy for this PR?

TODO

Are there any user-facing changes?

New ComposedNamedPhysicalExtensionCodec is added to the public API, while keeping the previous ComposedPhysicalExtensionCodec still available.

github-actions · 2026-08-31T17:06:19Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-proto v55.0.0 (current)
       Built [  60.036s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  58.164s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.127s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure type_marked_deprecated: #[deprecated] added on type ---

Description:
A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_marked_deprecated.ron

Failed in:
  Struct ComposedPhysicalExtensionCodec in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:1973

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [ 120.596s] datafusion-proto

gabotechs · 2026-08-31T17:05:23Z

+#[deprecated(
+    since = "56.0.0",
+    note = "Please use `ComposedNamedPhysicalExtensionCodec`"
+)]


Still not sure if it's worth to deprecate this one, FWIW both could live together.

gabotechs · 2026-08-31T17:06:22Z

+/// A PhysicalExtensionCodec that tries one of multiple inner codecs until one works.
+/// The name of the codec that successfully encoded an [`ExecutionPlan`] is stored in the
+/// encoded payload, and the codec with that exact name will be used for decoding.
+#[derive(Default, Debug)]
+pub struct ComposedNamedPhysicalExtensionCodec {
+    codecs: HashMap<Cow<'static, str>, Arc<dyn PhysicalExtensionCodec>>,
+}


cc @milenkovicm, do you have any opinions about this?

Could string make encoded message too big? Could you consider u8 keys

Or even u16 or u32, as the keys

If you consider sorted keys, you can implement codec priority as well. I know strings are more flexible than numbers but they won't remove synchronisation between orgs and they can be abused with key size

I don't think abuse is a worry here, as the provided String name is not un-sanitized input coming from the outside. Of course a developer can go crazy and provide an unreasonably long string here out of malice, but they can do that as well while serializing their custom plans, so it's not really a new risk.

My guess is that adding ~30 bytes worth of a string should not bee too bad, because this only affects custom nodes provided by users, not each node in the plan, so it should be negligible.

If you consider sorted keys, you can implement codec priority as well

🤔 this sounds interesting, but I don't understand it very well. How would this priority mechanism work?

coming from perspective of building binary protocols having string as a key does not look like best practice. you're right string keys might not be much but it does not mean it should be used, there is no real benefit of it compared to integers, apart from delaying decision on codec composition.

if you have sortable keys, you can add priority in which encoders should be executed, its a user decision, this makes encoding deterministic.

🤔 But determinism is not really the problem to solve here. For example:

Machine 1: [CodecA, CodecB, CodecC] <- pre-rollout
Machine 2: [CodecA, CodecD, CodecB, CodecC] <-post-rollout

Here, encoding is deterministic: codecs will always be attempted in the same order, but decoding still fails. It will deterministically fail every time until Machine 1 fully rolls out.

Maybe I'm missing how determinism can solve this issue?

I was talking about something slightly different.

Lets say you have event E which had 3 versions (lets say backward compatible), lets say that you have two codecs C1 and C2 each of which can encode E, C1 can encode in version 2, C2 can encode version 3. if you do not have codec priority, just by naming things, C1 could encode E even when C2 has latest version, so your encoding process will remove peace of information unnecessary.

so someone should make decision who to go first, and they should be able to say which one has precedence

👍 thanks for the explanation, I understand it now.

It sounds like just replacing the HashMap with a BtreeMap should be enough for addressing this. The BtreeMap will give this deterministic order in which codecs are tried for encoding.

codecov-commenter · 2026-08-31T17:13:34Z

Codecov Report

❌ Patch coverage is 0% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.56%. Comparing base (a274959) to head (d568122).
⚠️ Report is 205 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto/src/physical_plan/mod.rs	0.00%	97 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24826      +/-   ##
==========================================
- Coverage   81.58%   81.56%   -0.03%     
==========================================
  Files        1123     1123              
  Lines      406610   406707      +97     
  Branches   406610   406707      +97     
==========================================
- Hits       331719   331715       -4     
- Misses      55453    55554     +101     
  Partials    19438    19438

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gabotechs · 2026-10-02T09:39:19Z

Closing in favor of #24631

Add ComposedNamedPhysicalExtensionCodec

d568122

github-actions Bot added the proto Related to proto crate label Aug 31, 2026

github-actions Bot added the auto detected api change Auto detected API change label Aug 31, 2026

gabotechs commented Aug 31, 2026

View reviewed changes

gabotechs mentioned this pull request Sep 9, 2026

ComposedPhysicalExtensionCodec should not rely on positional encoding #24331

Open

gabotechs closed this Oct 2, 2026

Navigation Menu

Add ComposedNamedPhysicalExtensionCodec - #24826

Add ComposedNamedPhysicalExtensionCodec#24826
gabotechs wants to merge 1 commit into
mainfrom
gabotechs/composed-named-physical-extension-codec

gabotechs commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

gabotechs Aug 31, 2026

Uh oh!

gabotechs Aug 31, 2026

Uh oh!

milenkovicm Aug 31, 2026

Uh oh!

milenkovicm Aug 31, 2026

Uh oh!

milenkovicm Aug 31, 2026

Uh oh!

gabotechs Sep 1, 2026

Uh oh!

milenkovicm Sep 1, 2026

Uh oh!

gabotechs Sep 1, 2026

Uh oh!

milenkovicm Sep 1, 2026

Uh oh!

gabotechs Sep 1, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Aug 31, 2026 •

edited

Loading

Uh oh!

gabotechs commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gabotechs commented Aug 31, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabotechs Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gabotechs commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gabotechs Sep 1, 2026 •

edited

Loading

codecov-commenter commented Aug 31, 2026 •

edited

Loading